Text Embedding with Advanced Recurrent Neural Model

Embedding method has become a popular way to handle unstructured data, such as word and text. Word embedding, providing computational-friendly representations for word similarity, is almost be one of the standard solutions for various text mining tasks. Lots of recent studies focusing on word embedding try to generate a more comprehensive representation for each word that incorporating task-specific information. Yet compared with massive studies on word embedding, text embedding is still facing a lot of challenging problems. As “bag of embeddings” method is mainly used to generate text embedding, how to handle the order information in sentence becomes an important problem to be solved. Distributed representation, i.e. embedding, provides the possibility to encode the order information in a relative low-dimensional space, freeing the constraint of analyzing exponential states in sentence with traditional onehot representation. One promising approach to handle order information is recurrent neural network. Yet the original recurrent neural network and its extension, Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), still suffer the long-term gradient vanishing problem. Therefore long-term dependency or relationship is still hard to handle. In this project, we prosed two models, attention model and explicit memory model, that explicitly consider longterm dependency that try to solve the long-term dependency problem in text embedding. Experiments on real datasets illustrate their potential capacities to handle long-term dependency in text embedding task and related applications.